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ABSTRACT 


This paper provides a detailed description of how a smart spaces laboratory has been used for assessing learners’ 
performance in various educational contexts. The paper shares the authors’ experiences from using sensor-generated data 
in a number of learning scenarios. In particular the paper describes how a smart learning environment is created with the 
use of a range of sensors measuring key data from individual learners including (1) heartbeat, (ii) emotion detection, (iii) 
sweat levels, (iv) voice fluctuations and (v) duration and pattern of contribution via voice recognition. The paper also 
explains how biometrics are used to assess learner’ contribution in certain activities but also to evaluate collaborative 
learning in student groups. Finally the paper instigates research in the role of using visualization of biometrics as a 
medium for supporting assessment, facilitating learning processes and enhancing learning experiences. Examples of how 
learning analytics are created based on biometrics are also provided, resulting from a number of pilot studies that have 
taken place over the past couple of years. 
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1. INTRODUCTION 


For the past couple of years we have attempted to shift our work from investigating learner behaviour in web 
based e-learning communities to the investigation of collaborative learning with the help of sensor-generated 
data. This paper describes our experiences through a number of pilot studies where participants’ biometric 
data were collected for assessing the physiological and behaviour state of participants in collaborative 
learning scenarios. 

We argue that sensor-based collaborative learning provides an opportunity for increasing the 
effectiveness of dealing with social aspects in education. By reaching concrete findings on the way learners 
are affected in certain learning scenario we can safely make further assumptions on how e-learning affects 
learners. We could also be able to safely propose classifications of learners according to dominant patterns in 
the way participants are affected during different learning activities. We also argue that sensor-based data 
collection could enhance student learning. Students can become familiar with such observation and data 
collection techniques, while appreciating the use of data visualization, learning analytics and the impact of 
intelligent environments in educational contexts. This approach in educational technology could allow 
self-assessment in the form of profiling individual learners based on their biometric data, in a way similar to 
the profiling taking place in e-learning platforms and virtual learning environments. Our views are advocated 
in the relevant literature, as Ara et al (2012) propose a sensor-based project management process, which uses 
“continuous sensing data of face-to-face communication, was developed for integration into current project 
management processes”. Therefore, this work is primarily focused on learning activities that are in line with 
project management scenarios. 
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Previous work focused on introducing multi-sensor settings for observing social and human aspects in 
project management (Dafoulas et al, 2017). The work considered the use of sensors in identifying patterns of 
collaboration in Global Software Engineering student teams engaging in activities involving brainstorming, 
task management, decision making and problem-solving. Emphasis was given on identifying the range of 
sensors that could be used to extract useful findings while learners engaged in team activities. The 
preliminary findings demonstrated that the data generated by sensors during learning activities could be used 
to deduct useful findings in relation to individual and team performance. Additional work focused on aspects 
associated with the infrastructure needed to establish ‘smarter’ teaching environments fostering collaborative 
hybrid learning (Dafoulas et al, 2016a). The outcome of this work was a reflective discussion on the role of 
different sensors in collecting information relating to learner’s biometrics during learning activities. Further 
work focused on the role of dashboards in visualising global software development patterns (2016b). 

Following the findings of previous work, the research study concentrated on setting up a smart learning 
environment that would enable to investigate physiological and behavioural aspects of learners during certain 
activities. The study is based on the hypothesis that the collection of appropriate biometrics and the analysis 
of physiological and behavioural patters during a learning experience can help introducing appropriate 
interventions for enhancing the learning experience. The study also focuses on exploring possibilities of 
using biometrics as additional means for evaluating learners’ state during specific learning activity types and 
perhaps considering certain biometric analysis for assessment and feedback purposes. For example the 
physiological aspects of learners could focus on facial expressions and the existence of certain emotions in 
association to progress, challenge and collaboration during the learning experience. Instructors could use 
emotion recognition as a tool for identifying tasks that require additional support for students. On the other 
hand, voice recordings can be used to assess behavioural patterns, showing whether learners demonstrate 
different attitudes for different learning activities. 


2. SETTING UP A SMART LEARNING ENVIRONMENT 


Our work is carried out in a specialist laboratory called smart-spaces lab. This is a facility equipped with a 
range of sensors and is laid out as a small home where activities can take place in a living room, a bedroom, a 
couple of office-type rooms, while there is a kitchen and a restroom. It was important for our study to be 
conducted in a dedicated space, where any type of disruption that is possible in traditional classrooms would 
not affect participants. The participants’ familiarisation with the learning space occurred one week prior to 
the pilot studies, where they were introduced to the building and were given a chance to understand the tasks, 
as well as the different use of the space. Learners were also introduced to the different sensors that would be 
used, and detailed explanation was used on the nature of the study, the data collected, the nature of 
information gathered and a detailed description of how the collected data would be analysed. 


2.1 Identifying Learning Activities suitable for Smart Learning Spaces 


It was important to identify suitable learning activities that would help extract meaningful results from the 
collection of biometrics. The nature of the data collected is such that would be best interpreted in association 
to scenarios where participants would interact with each other and engage with specific tasks. While several 
works focus on collaborative technologies in education such as virtual learning environments, synchronous 
and asynchronous communication, computer aided learning and establishment of online learning 
communities, these are not necessarily suitable for collecting and analysing biometrics effectively. 

It is important to identify the scope of such study, which is to assess how physiological and behavioural 
states of learners can help evaluating collaborative learning. Therefore it is imperative to identify activities 
that would ensure participants engagement in certain ways. For example individuals attending a lecture are 
unlikely to demonstrate significant changes in their state over a 45-60 minute period, while their involvement 
with a decision-making task may demonstrate changes in their behaviour or even well-being. Another 
example is the collaborative nature of certain learning activities, as it is anticipated that participants are more 
likely to go through more dramatic changes in their state when they are part of a team that requires 
interaction between its members and synchronous communication. Finally, the type of data collected (i.e. use 
of sensors in a controlled setting) meant that the tasks had to be designed in a way that participants would be 
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positioned in a way allowing them to perform their tasks while allowing the unobstructed collection of the 

necessary biometrics. 

We have decided that primarily two types of learning activities would allow us to reach some conclusive 
findings in relation to the way physiology and behaviour of learners change over a learning process. These 
were (i) meeting activities and (ii) presentation activities. The former type of learning activity would allow 
student to engage in brainstorming, decision-making, problem solving and reaching agreement. The latter 
type of learning activity involves presentation skills, communication and coordination of the task, as well as 
planning and management of the presentation. 

For the purpose of this study we have identified a number of key concepts for investigation. Each of the 
concepts is associated with particular learning activities as discussed below: 

e Duration — associated with the time it takes for a team to reach consensus. The duration of the 
brainstorming and decision making activities is recorded for assessing whether teams with members 
expressing certain physiological state (e.g. increased frustration or sadness expressed in their facial 
expressions) or behaviour (e.g. stress represented in high pitch voice or raised tone) are likely to take 
longer to reach consensus. 

e Participation — associated with the involvement of individual members in team communication. The 
participation of each member is recorded as a series of coded actions observed in video recordings of 
team meetings. Examples may include members probing for answers the rest of the team, introducing 
own ideas or taking initiative during the discussion for facilitating the team decision-making process. 

e Presentation — by determining the proportion of each member’s part in the team presentation we can 
determine whether all members have contributed an equal portion of the team effort. This concept 
attempts to use the data gathered by recoding members voice to identify unbalanced contributions to the 
team presentation and facilitate more accurate and fair assessment. 

e Concern — by monitoring biometrics (e.g. heart beat, sweat levels) it is possible to make associations 
between stress levels and certain physiological states for certain participants in team decision-making. 

e Disagreement — associated with the observation of discussion debates and difference of opinion. This 
concept attempts to identify whether certain biometric data patterns are related to conflicts in team 
communication and collaboration. 

e Emotion — associated with a range of emotions during specific meeting milestones. For example the 
recognition of certain emotional states during the brainstorming session can help to identify any 
association of certain physiological states and individual performance. 

e Contribution — the contribution of team members in activities is recorded in the form of individual effort 
towards teamwork. This can be observed as coded behaviour during decision-making and problem 
solving activities. 

e Misalignment — by identifying differences between individual and team decisions during the 
decision-making activities it may be possibly to creation association between behavioural and 
physiological states and states of team performance or even team member progress. 

The following section discusses how these concepts are monitored during the learning scenario of this 
research study. There are distinct phases in the study aiming to provide clear milestones for the learning 
process. Prior to the discussion of the method followed, the paper discusses the selection of appropriate 
sensors for assessing the role of biometrics in education. 


2.2 Selecting Appropriate Sensors 


There was a range of sensors that we considered in an effort to select the most appropriate ones for the 
purpose of this study. Participants were allowed to use different areas of the lab in order to reflect on their 
individual contributions and fill in their forms without much obstruction or interference from other team 
members. Students then moved to one of the meeting areas where the sensors were placed around the 
meeting table. Finally the presentations took place in the living room area. 

Several sensors and data collection techniques were considered in preparation for the pilot study. Heart 
beat monitors were used in certain sessions of both pilots. There is uncertainty whether significant findings 
can be retrieved from these, as there is no sufficient time to monitor individual heart rate patterns in order to 
provide a comparable baseline. Galvanic Skin Response sensors were considered for monitoring perspiration 
as an alternative to heart beat monitors that were suitable for the team meeting setting, as there was minimum 
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movement across the room. During the first pilot the GSR sensors were connected to individual computers, 
while the second pilot study involved the creation of a self-contained multi-sensor allowing participants to be 
more mobile. The use of a portable polygraph to record physiological measures such as respiration, pulse, 
blood pressure and skin conductivity, although very useful was considered as intrusive and impossible to use 
without affecting the project meeting process. Heat sensor cameras were also considered for the presentation 
recordings and it was decided for use in a follow up pilot. Another possible source of data was the use of eye 
tracking software for measuring blink rate and direction of gaze. It was decided that this sensor would be 
better used for a scenario that would require one-to-one communication where two members would face each 
other when negotiating. Furthermore, the fact that all members would be sitting around a meeting table meant 
that the use of a sociometer for measuring cooperation and collaboration in physical space would not be of 
use for the specific pilot. Researchers at MIT have worked on “using statistical pattern recognition techniques 
such as dynamic Bayesian network models we can automatically learn the underlying structure of the 
network and also analyse the dynamics of individual and group interactions” (Choudhury and Oentland, 
2003). This work provides an excellent opportunity for understanding the structure of face-to-face 
interactions and possibly to replicate similar models for online interactions. The authors have identified the 
opportunity to use heat sensors in the third pilot during the newly introduced stage of scrum meetings as 
discussed later in the paper. The sensors and recording devices used for the collection of data during the pilot 
study focused on the collection of video, audio, and physiological data. Emphasis was given on identifying 
the extent of individual contribution, while assessing individuals’ emotion. Future plans include further 
analysis of the sensor data with tagging of the meeting videos. 

Audio data was collected using the Kinect for Xbox One, motion-sensing device. The device was 
programmed to collect the sound source angle (in degrees), and the direction that sound is arriving from a 
sound source. This enabled to observe the verbal participation of the individuals in the group. Audio data 
was collected during the team decision-making stage and presentation stage. This allowed determining 
individuals’ participation in the team meeting and the proportion of the presentation each member delivered. 
Our objective was to investigate whether team cohesion could be determined from team member contribution 
using audio behaviour. This would be in line with existing work on analysing group behaviour within the 
context of cohesion and especially automatically estimating high and low levels of group cohesion (Hung and 
Gatica-Perez, 2010). 

Electrical conductance of the skin was measured using galvanic skin response sensors. Six such sensors 
were built, one for each team member. Each participant had to hold the sensor during the entire project 
meeting (on their thumb) measuring physiological and emotional arousal. This type of sensor, although it has 
received criticism on the merit of its accuracy and the ability to provide an acceptable measure of stress, has 
been used in the past (Villarejo et al, 2012). Our objective was to investigate possible association from 
different conductance of the skin at certain points of each meeting that could be perceived as stressful. 

Further to the previous two sensors, each participant had a camera focusing on their facial expressions, 
connected to a laptop. Participants’ facial expressions were collected in real-time during the decision making 
process. Team members’ facial expressions were collected as input and returned set of emotions for each face 
as well as the bounding box for the face using Microsoft Face API. The possible emotions to be detected 
were anger, contempt, disgust, fear, happiness, neutral, sadness and surprise, which are universally 
communicated using facial expressions. During this preliminary investigation the team did not focus on the 
selection of different algorithms for the analysis of emotions, as the scope was to investigate the usefulness of 
this data stream rather than the accuracy of the emotion of each subject. Further work is planned to determine 
the level of accuracy for emotion detection based on specific algorithms. The objective was to associate 
expressions to particular emotions of team members during certain points during the decision making process 
and throughout the consensus meeting. In the past robust recognition of facial expressions from images and 
videos was still a challenging task due to the difficulty in accurately extracting emotional features (Zhang and 
Tjondronegoro, 2011). However, “significant performance improvements due to the consideration of facial 
element and muscle movements” have improved the performance of facial expression recognition systems. 
Based on the preliminary data analysis we have conducted we are working towards using the same technique 
during video conferencing between remote team members when collaborating in scenarios similar to the one 
described in the first pilot study. 

Using a camcorder video and audio of the group presentation was recorded and several pictures were 
taken. The objective was to tag videos for synchronizing the data collected from various sensors and also 
identify important milestones during the decision-making and presentation stages. Any notes, which 
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participants had written on the given forms during the experiment were collected, to analyze input individuals 
had on the final decision. Figure | shows the setting of the decision-making project meeting and the set-up of 
the sensors, recording devices and laptops. 


3. SUGGESTED METHOD FOR USING BIOMETRICS IN EDUCATION 


As mentioned earlier, the research study is based on a learning process scenario that consists of distinct 
stages. During the past two years, two pilots have been conducted with two sessions for each pilot. The first 
year included three stages in the learning process, while in the second year an additional stage was 
introduced. The learning scenario involves student teams that are prepared for a presentation to describe their 
progress in strategic management of information system. The assessment is on their efforts to provide 
consultancy to a real organisation that they have interviewed with focus on how the business will benefit 
from the introduction of social networks in key areas such as Customer Relationship Management, 
e-Commerce and Supply Chain Management. Each team consists of up to six members and each member is 
given certain tasks to perform. The team report includes a total of eighteen sections, with certain members 
being responsible for a certain number of these sections. We will discuss next the way the research pilot is 
organised. In all pilots, participants were informed in detail about the observation techniques that the 
different approaches followed for collecting biometric data. Each participant provided a consent form, after 
going through the report on the way data is collected, used and stored. The first pilot was structured around 
three main stages. These were as follows: 

e Stage 1 — focusing on determining individual preferences. During the first stage of the pilot study, 
emphasis is given on identifying each participant’s perspective prior to a project management meeting 
that will aim to reach consensus. Each member must decide from a list of eighteen topics the number of 
topics that can be presented in a period of 90 seconds. Each member needs to identify how many topics 
would be realistic to present in the given time, as well as those topics that he/she could cover. 

e Stage 2 — focusing on team coordination. The second stage of the pilot involves all team members in a 
coordination activity. A project manager is appointed who needs to align the suggestions of individual 
team members and facilitate the team in reaching consensus. The Each member uses the form where the 
suggested topics for presentation are identified to explain his/her views to the rest of the team. The 
decision making process needs to end with team consensus, and the project manager is not allowed to 
make decisions on behalf of the group. 

e Stage 3 — focusing on team presentation. The final stage involves all members of the team in a 
presentation, where each participant is responsible to deliver the topics that have been assigned to 
him/her during the project management meeting. The scope of the presentation activity is to assess the 
balance of contributions from all team members. 


Table 1. Collecting biometric data in the first pilot study 


Learning process Biometrics collected Associated concepts 


Stage 1 N/A N/A 

Stage la Emotion recognition Concern / Disagreement / Duration 
Voice recognition Participation / Contribution 
Heart beat Concern / Disagreement / Emotion 
Sweat level Concern / Disagreement / Emotion 

Stage 2 Emotion recognition Concern / Disagreement / Duration 
Voice recognition Participation / Contribution 
Heart beat Concern / Disagreement / Emotion 
Sweat level Concern / Disagreement / Emotion 
Behaviour observation Emotion / Misalignment 

Stage 3 Voice recognition Participation / Presentation 


Behaviour observation Duration / Emotion 


Table 1 shows the different biometrics collected during the three stages of the first pilot study (excluding 
text in bold describing stage 3a). The first stage of the experiment requires participants to fill in their forms; 
therefore no biometric data were collected. However, the second stage involved (i) emotion recognition of 
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each participant during the decision-making process, (ii) voice recognition in the form of duration of 
contribution in relation to other participant (i.e. how long each member talked during the meeting), (ii) heart 
beat fluctuation over the ten minute meeting, (iv) sweat levels during the meeting, (v) changes in the pitch or 
volume of the voice, (vi) body language during the experiment tagged in video recordings. The third stage of 
the study involved the recording of changes in the voice’s pitch and volume as well as video recording for 
tagging behaviour of each presenter. Following the experience from the first pilot sessions, it became evident 
that the study required an additional stage for collecting individual biometrics. Furthermore the pilot sessions 
followed slightly different scenarios investigating differences in reaching consensus when sub-teams are 
involved in the process. The scope was to evaluate how team structure complexity affects performance and if 
this is evident from the collected biometrics. 

The second pilot introduced a number changes, with the main one being the introduction of another stage 
before the project management meeting. Once all participants filled in the forms with their views on which 
topics should be presented, then a scrum-type meeting was organised. This is a technique used in agile 
software development that requires team members to participate in a stand-up meeting in order to share their 
progress with their team. During this pre-meeting session, each team member was required to briefly explain 
their contribution to the team’s work so far and explain their rational for selecting the topics to be presented. 
This offered the opportunity for the team to find out the strengths and weaknesses of each member and also 
appreciate the reasoning behind the different views that would have to be assessed in an effort to reach 
consensus. As shown in table 1 (in bold) the new stage introduced the opportunity to record four different 
biometric data. During the scrum meeting, the entire team was video recorded, while each member had their 
faces recorded during their brief stand-up talk. These video feeds can be used for emotion recognition of 
individuals, as well as behavioural patterns of other team members while listening to each member’s 
progress. The team members had their hart beat and sweat levels monitored during the scrum meeting, with 
emphasis on changes to their behaviour and physiological state during their own stand-up talk. 

It is important to state that the second pilot also involved an additional feature adding complexity to the 
task. While the first session was based on the participation of single teams, the second session involved the 
collaboration between sub-teams or pairs of participants from the original teams. Each team of students after 
the first twelve weeks of the course was split into pairs or sub-teams consisting of three members. These 
sub-teams had to produce the same reporting for the final part of the course. Therefore, during the second 
session of the pilot, these sub-teams had to share their progress with each other before the project 
management meeting. This was achieved with the introduction of the scrum meeting and the brief stand-up 
talks. 


4. LEARNING ANALYTICS OF SENSOR-GENERATED BIOMETRICS 


As mentioned earlier in the paper we have created a smart spaces laboratory being used for collecting 
biometrics during learning experiments. In this section we will present the layout of the learning space, we 
will present some of the findings and discuss some aspects of the produced learning analytics. The 
photographs included in figure 1, show how the experiment took place. The top-left picture is a 
demonstration of the study’s concept when video recorded for promotional purposes. The entire equipment 
used including the sensor apparatus are on display. The top-right and bottom-right photographs show the 
consensus meeting setting for the first and second pilots respectively. Emphasis is given to show how the 
emotion recognition processing takes place with the use of individual web cameras (top-right photograph) 
and the use of Kinect to determine voice patterns and participant contribution (bottom-right photograph). The 
final photograph shown at bottom-left shows the team presentation and the use of a video camcorder for 
recording any behavioural patterns, as well as the Kinect used for recording presentation balance and 
participation from all team members. 
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Figure 1. Smart learning space lab set-up 


Figure 2 shows the compiled biometrics in a single dashboard. The scope of this graph is to show the 
volume of information that is generated in a single consensus meeting and associated presentation for a team 
of five participants. The first graphs located at the top of the dashboard show how each member participated 
in the discussion, with emphasis on the shift between different speakers. The bard graphs following next 
show the average and total time each member contributed. This top part of the dashboard shows the 
biometrics for the consensus meeting to the left and the corresponding data for the presentation to the right. 
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Figure 2. Dashboard compiling all biometric data per participating team 


The bottom left part of the dashboard provides the emotion recognition for each member with a specific 
timestamp for each different emotion or facial expression identified. For example there are clear timestamps 
showing how long each member expressed certain emotions based on the six basic emotions including 
(1) anger, (ii) disgust, (ii1) fear, (iv) happiness, (v) sadness and (vi) surprise. Finally, the bottom right part of 
the dashboard showed the sweat levels for each member at different timestamps and the average value for 
each participant. A series of very interesting findings have been identified following further analysis of the 
biometrics as we can see from the follow figures. In figure 3 we can see the classification of participant 
contributions according to the part of the consensus meeting they have contributed the most. Each participant 
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is represented with a single bar and the fourteen groups are clearly visible. Participants in red have 

contributed mostly in the first quarter of the meeting (start), while participants in blue contributed mostly 

during the last quarter of the meeting (end). Finally the participants shown in orange provided their 

contribution during the middle of the meeting (the second and third quarter of the meeting together). The key 

findings from this analysis are as follows: 

e 79% of learners, who contributed more that the average participant, provided their contribution at the 
middle part of the meeting. 

e 92.9% of learners, who contributed the most in comparison to the rest of their team, provided their 
contribution at the middle part of the meeting. 

e 72.7% of learners, who contributed most at the start of the meeting, seem to participate below the 
average participant in all teams. 

e 92.9% of learners, who contributed most at the end of the meeting, also show smaller contribution in 
comparison to the average participant. 

There seems to be a pattern with students who are les likely to contribute in the discussions to be vocal 
either at the beginning of the meeting, usually by presenting briefly their views or at the very end of the 
meeting to confirm agreement with the decision. Most prominent contributors tend to dominate the middle 
part of the meeting and the discussions leading to consensus. 
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Figure 3. Classification of participant contribution during the consensus stage 


Again as in the previous figure, the same colour scheme is applied to show those participants who 
contributed more during the first quarter (start) of the presentation (red colour). Those participants who 
contributed during the second and third quarters (middle) are shown in orange and the participants who 
contributed towards the end (last quarter) shown in blue. The horizontal line shows the average contribution 
by the participating learners. The main findings are as follows: 

e 83.3% of learners, who participated in the presentation above the average, do so during the middle of the 
presentation. 

e 81.8% of learners, who participated at the start of the presentation, show below average participation. 

e 92.9% of learners, who participated at the end of the presentation, show below average participation. 

It appears that the vast majority of the teams positioned their key contributors in the middle of their 
presentation. This is advocated by the observation of the video recordings for each of the participating teams. 
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Figure 4. Classification of participant contribution during the presentation stage 


5. CONCLUSION 


In this paper we discussed our approach in establishing a smart learning space in order to observe learners 
engaging in a range of learning activities. Our work focuses on collecting and analysing biometrics in an 
effort to determine certain physiological and behavioural states associated with learning experiences of 
individuals and teams. The paper discussed the method used, presented the techniques for collecting 
biometrics and briefly presented some key findings from analysing part of the collected data focusing on 
voice recordings. 
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